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APOLLO EXPERIENCE REPORT 

RELIABILITY AND QUALITY ASSURANCE 

ByK. P. Sperber 
Lyndon B. Johnson Space Center 

SUMMARY 


The reliability and quality of the Apollo spacecraft and associated equipment re- 
sulted from the application of reliability and quality assurance practices and from the 
techniques developed and adopted during the engineering and manufacturing phases. 

Two reliability and quality practices adjusted as the program developed included: (1) the 
establishment of NASA Lyndon B. Johnson Space Center (formerly the Manned Space- 
craft Center) and contractor reliability and quality assurance management review meet- 
ings, and (2) specifying in the contract the scope, format, and content of reliability and 
quality plans. 

A contractor parts program was implemented to increase the integrity of the 
equipment. This program included requirements for parts derating and screening, 
control of the use of nonmetallic materials in the spacecraft, procedures and controls 
for assuring cleanliness of both the equipment and fluid systems, and a comprehensive 
testing program. These engineering and manufacturing techniques were developed as 
the program progressed. 

The more significant experiences are presented in this report as guidance for 
future space-flight programs. The experiences described indicate in many specific 
ways deficiencies that were detected and corrected to improve the reliability and quality 
of the equipment. 


INTRODUCTION 


A summary of the reliability and quality assurance (R&QA) experiences in the 
Apollo Program is presented. The report was written primarily to be used by Gov- 
ernment and industry program managers who contribute to the design, fabrication, 
testing, and operation of safe and reliable equipment for space applications. Reliabil- 
ity and quality are presented as product attributes, not as organizations. 



MANAGEMENT OF RELIABILITY AND QUALITY ASSURANCE 


The basic policy for the R&QA activities in NASA programs establishes the re- 
sponsibilities for NASA program directors and managers. This policy provides for 
the planning, organizing, conducting, and evaluating of the activities to ensure the re- 
quired levels of reliability and quality. The R&QA activities are intended to ensure 
that the equipment produced for the Government is in accordance with the requirements 
of the contract; however, the R&QA functions performed by the Government do not in 
any way relieve contractors of the responsibility for the integrity of equipment or 
services. 

During the early developmental phase of the Apollo Program, some R&QA per- 
sonnel were organizationally a part of the Apollo Spacecraft Program Office (ASPO) 
while the rest of the R&QA personnel were decentralized. These personnel that were 
not a part of ASPO and their associated R&QA effort were a part of other Lyndon B. 
Johnson Space Center (JSC), formerly the Manned Spacecraft Center (MSC), organi- 
zational elements such as the Flight Safety Office, technical divisions of the Engi- 
neering and Development Directorate, or the Resident Apollo Spacecraft Program 
Office (RASPO) that was established at the prime contractor plants. 

The MSC Flight Safety Office (with specialized safety, reliability, and quality 
personnel) provided assistance to the ASPO and RASPO elements during program de- 
velopment. This office also performed the inspection function for all equipment fab- 
ricated, assembled, and tested at JSC. The R&QA personnel associated with the 
technical divisions ensured the implementation of R&QA requirements in the contracts 
for which the divisions were responsible. The RASPO R&QA elements performed re- 
quired functions associated with the contractors' R&QA effort. 

This decentralization of R&QA functions and responsibilities between these JSC 
elements resulted in differences regarding the establishment and interpretation of re- 
quirements, the degree of implementation, and the monitoring of contractor R&QA 
activities. These differences occurred because of the variances of opinions between 
the personnel in the organizational elements. Program-oriented personnel have a 
different philosophy about R&QA disciplines than the R&QA management and policy 
personnel have. 

In 1968, all but two groups of the R&QA elements were reorganized into one 
central R&QA office responsible for all R&QA activities associated with all spacecraft 
hardware and providing appropriate support to all program offices and JSC organiza- 
tional elements. The two groups not included in the reorganization were the Mission 
Control Center of the Flight Operations Directorate and the Aircraft Operations Office 
of the Flight Crew Operations Directorate. This centralization aided in establishing 
coordinated requirements and provided for the uniform interpretation and implementa- 
tion of the R&QA tasks including the monitoring activities. 

To achieve the most uniform and useful implementation of R&QA policies, the 
R&QA should be an integrated, centralized organizational element reporting to the 
Center Director. This organizational element should have defined functions stating 
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its responsibility for all R&QA activities. To be effective in contributing to the 
overall goals of a program, the organization should be both hardware and problem 
oriented; have a close working relationship with program elements such as engineer- 
ing, design, and procurement; provide rapid response capability in all technical skill 
areas; and have direct access to top management. 

The management of R&QA functions consists of the establishment of R&QA 
requirements and assessment of the R&QA functions that implement the requirements 
and of the deliverable equipment to ensure continued effectiveness. To be effective, 
the functions must be planned, continually monitored, and adjusted as the program 
advances. 


Reliability and Quality Assurance Requirements for 
Space Systems Contractors 

Background. - The R&QA requirements are disciplines that, if properly imple- 
mented by appropriate elements of a contractor organization, will enhance the proba- 
bility of producing and delivering reliable equipment. The reliability requirements 
are defined in NASA Handbook 5300. 4(1A), "Reliability Provisions for Aeronautical 
and Space System Contractors, " April 1970. The quality requirements were published 
in NASA Handbook 5300. 4(1 B), "Quality Program Provisions for Aeronautical and 
Space System Contractors," April 1969. These requirements include the activities 
that should be implemented for major aerospace systems such as the Apollo Program 
and are the result of inputs solicited from NASA centers, other Government agencies, 
and industry. Much experience gained from the Apollo Program and from other aero- 
space programs is reflected in the publications. In addition, the NASA R&QA publica- 
tions include associated efforts to be performed by the engineering, manufacturing, 
purchasing, logistics, and program- management elements of a contractor. 

Experience . - The JSC experience indicates that the R&QA requirements should 
be selected or "tailored' ' to meet the needs of the procurement and to ensure achieve- 
ment of the desired discipline at the lowest possible cost. This tailoring is accom- 
plished by using the basic requirement document as a guideline. The requirements 
selected are included in purchase orders and contract statements of work only to the 
extent necessary as determined by an evaluation of the equipment criticality, charac- 
teristics, and end use. The evaluation is performed jointly by the technical monitors 
and R&QA personnel. This tailoring concept was developed as the Apollo Program 
progressed. 

Reliability requirements . - The requirements for reliability are grouped into 
reliability management, reliability engineering, and testing and reliability evaluation. 
The management of reliability functions is most effective when resources are directed 
to the following tasks. 

1. Review of design specifications 

2. Review of failure mode and effects analyses (FMEA) 

3. Review of single failure point (SFP) summaries 
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4. Review of maintainability analyses and maintenance plans 

5. Participation in design reviews 

6. Participation in the investigation and analysis of failures and other problems 

7. Review of proposed corrective action resulting from participation in the in- 
vestigation and analysis of failures and other problems 

8. Assessment of the use of electrical, electronic, and electromechanical 
(EEE) parts 

9. Control of materials specified by designers 

1 0. Assessment of test plans, procedures, and results 

Quality assurance requirements . - The quality assurance requirements are 
grouped into the following major activities. 

1 . Quality assurance management and planning 

2. Design and development controls 

3. Identification and data retrieval 

4. Procurement controls 

5. Fabrication controls 

6. Inspections and tests 

7. Nonconforming article and material control 

8. Metrology controls 

9. Stamp controls 

10. Handling, storage, preservation, marking, labeling, packaging, packing, 
and shipping 

11. Sampling plans, statistical planning, and analysis 

12. Government property control 

The JSC experience in the management of contractor R&QA functions indicates that, 
in most procurements, some of all the preceding requirements are needed for a well- 
managed program. These requirements are not concentrated in a single R&QA orga- 
nization and are intended for use by those contractor elements responsible for the 
design, development, fabrication, assembly, and testing of the equipment. By par- 
ticipating in the evaluation of contractor proposals before contract award, the R&QA 
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Office can advise program managers of any excessive or unnecessary cost items and 
of the R&QA effort to be implemented by the contractor. 

After a contract award, contract changes that affect the deliverable equipment 
are reviewed by the R&QA Office to ensure that the original R&QA requirements ne- 
gotiated in the contract are still valid. In addition, the contractor R&QA tasks are 
continually monitored to ensure that the effort is maintained at the proper level. 


Government Source Inspection 

Background . - The NASA requires the contractor quality organizations to review 
procurement documents before release to ensure that appropriate quality assurance 
requirements have been included. The NASA quality assurance representative is re- 
quired to review the procurement documents to determine if Government source in- 
spection is necessary and to prepare a letter to the Government agency at the supplier 
facility delegating quality assurance functions to that agency. 

This review at the prime contractor plant slowed down the release of the pro- 
curement documents to an unacceptable flow. This reduction in flow was caused by 
limited NASA R&QA staffing for the review. The procurement document release 
process was enhanced by having the prime contractor R&QA personnel review and re- 
lease the documents using the guidelines prepared by the JSC R&QA. The Govern- 
ment review of the documents was made after their release. The JSC experience may 
be summarized as follows. 

The contractor and NASA R&QA organization must have adequate staffing for 
review of a large number of procurement documents early in the procurement cycle 
for large contracts. In conjunction with this, the program management must allot a 
reasonable time for this procurement review or risk hardware delivery and quality 
control problems later. 


Lyndon B. Johnson Space Center Procurement of Space-Flight Equipment 

Background . - The Government-furnished equipment (GFE) for the Apollo Pro- 
gram consisted initially of the guidance and navigation system, the flightcrew suits, 
the portable life-support system, and miscellaneous other items. For each of these 
procurements, the JSC R&QA Office elements adequately supported the JSC technical 
personnel. As additional GFE and new experiments were authorized for the Apollo 
missions, the number and the rate of initiation of GFE procurements increased 
significantly. 

Experience . - As the additional GFE and experiments contracts increased in 
number and began reaching the hardware stages, it became apparent that the limited 
JSC R&QA staff could not monitor and support each contract to the extent applied to 
the spacecraft hardware. A decision was made to concentrate the R&QA efforts on the 
critical hardware (such as flight cameras, lunar experiment equipment, communica- 
tion headsets, and others) and to provide only minimal surveillance to the rest except 
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for significant problems. The hardware contractors were thus required to provide 
the maximum available R&QA functions with only a relatively small amount of Govern- 
ment direct support. 


Quality Assurance Functions of Government Agencies 

Background. - Quality assurance functions are performed by other agencies of 
the Government under delegations issued by the appropriate NASA or JSC contracting 
officer. These agencies, which include the Air Force, the Army, the Navy, the 
Atomic Energy Commission, and the Defense Contract Administration Service, are 
referred to as other Government agencies. 

Contract administration services costs (including quality assurance) are reim- 
bursable under NASA/Government-agency agreements. When the NASA work is dom- 
inant or extensive, NASA or JSC R&QA personnel are placed in the contractor 
facilities to assist and provide technical direction regarding quality matters to the 
Government agency. 

Experience in the selection of Government quality assurance functions . - For 
reasons of cost effectiveness, the practice of tailoring the R&QA requirements to the 
procurement was established at JSC . The procurement documents are reviewed by 
reliability and quality engineering personnel for equipment complexity, criticality, 
special processes, test requirements, and value. From this review, the quality func- 
tions to be performed by the Government agency are selected or tailored from the 
basic requirement document (NASA Handbook 5300. 4(2B), "Quality Assurance Provi- 
sions for Government Agencies, " November 1971 edition) and forwarded to the contract- 
ing officer for transmittal to the delegated agency. 

Typical functions performed by the Government agencies for JSC are inspection 
of equipment and incoming material, witness tests, failure notification to JSC, review 
of nonconforming equipment, and surveys of suppliers quality assurance operations. 

At the beginning of the Apollo Program, the delegation of quality assurance functions 
to the Government agencies caused some concern because of the difference between 
the JSC requirements and the procedures normally implemented on Department of 
Defense (DOD) and other Government contracts. The JSC quality assurance repre- 
sentatives were designated to assist the Government agencies to ensure that the re- 
quirements were understood and implemented properly. The JSC experience indicates 
that the services of Government agencies can be used most effectively if timely quality 
direction and assistance are given and points of contact are provided. This direction, 
which is provided by the JSC quality assurance representative, includes interpreting 
R&QA requirements, evaluating the adequacy of fabrication and inspection processes, 
providing guidance in the preparation of procedures, assisting in the disposition of 
nonconforming equipment, and resolving any problem or question the Government 
agency may have in performing the delegated functions. 


Documentation Requirements 

General. - The planning and assessment of R&QA functions requires the receival 
and review of several types of documents. Initially, documentation as required in the 
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NASA R&QA publications was requested from the contractors without any additional 
documentation criteria such as format, scope, and content. It soon became evident 
that the documentation requirements must be adjusted and tailored for each procure- 
ment in a similar manner as the R&QA requirements, and that the scope, format, and 
content of documentation must be specified in the procurement documents. Examples 
of documents used by the JSC R&QA organization include reliability and quality plans, 
EEE parts specifications, parts list, certification and acceptance test plans, failure 
reports, and failure analyses. Except for the R&QA plans, the other documents listed 
are required by JSC engineering and program office elements. 

Background - R&QA plans . - The Apollo spacecraft contractors were required 
to prepare R&QA plans and submit them to JSC for approval. This requirement was 
new to the aeronautics industry. Quality control system requirements imposed on 
DOD contractors require them to document the system including procedures subject 
to disapproval by the Government. Submittal of this document is not required but it 
must be available to the Government. 

Experience - R&QA plans . - In the judgment of the JSC R&QA personnel who 
assisted the contractor in the development of the R&QA plans, the plans constitute a 
good medium for contractor and Government management to understand the objectives, 
procedures, and milestones of the R&QA functions. For maximum benefits, this same 
discipline should be used by the prime contractor in the review and approval of sub- 
contractor plans. 

Experience in the area of documentation has shown that the scope, format, and 
content of R&QA plans must be specified in the contract to preclude arbitrary changes 
in acceptance criteria. To be fully effective, these plans must be established in the 
contract as documents requiring customer approval. The JSC R&QA personnel have 
found that R&QA plans are most beneficial on complex or costly procurements. Tech- 
nical and financial risks are the greatest for equipment the design of which is new or 
unproved, especially for costly and very complex hardware. For these types of pro- 
curements, the submittal and implementation of R&QA plans help provide confidence 
and assurance that technical problems and costs associated with these risks are mini- 
mized. It was found that normally no benefits were obtained by requesting the develop- 
ment and submittal of plans for short-term, less costly, off-the-shelf procurements. 

R&QA status reports . - In the early stages of the Apollo spacecraft development, 
major contractors were required to submit periodic reports on the progress of the 
reliability and quality assurance activities. Scope, format, and content of these re- 
ports were determined by the contractors. These status reports proved to be ineffec- 
tive because the program was moving ahead rapidly and the data in the reports were 
not current and not in sufficient detail to permit effective assessment of the R&QA 
activities. The Apollo prime contractors were directed to discontinue the status re- 
ports and to prepare monthly briefings for JSC R&QA management personnel. These 
briefings were given by the contractor R&QA managers to the JSC R&QA manager and 
division chiefs. The meetings were conducted in accordance with an established agenda 
and were documented. Action items were established, and handout material was pro- 
vided for the participants. Topics normally discussed were budget control (including 
manpower and costs), supplier hardware problems, quality trend data showing defects, 
rework, material review actions, failure information, and corrective actions taken to 
resolve problems. 
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These management review meetings are effective, especially in the early phases 
of a program. The NASA personnel from Washington, the John F. Kennedy Space 
Center (KSC), MSC, the White Sands Test Facility, and the quality assurance offices 
resident at prime contractor plants participated in the briefings, providing valuable 
viewpoints from several perspectives. Many briefing topic problems, which would 
have taken considerable time to resolve if correspondence or telephone conversation 
media had been used, were discussed at the briefings. Action was assigned and, in 
most cases, the problem was resolved before the next regular meeting. These meet- 
ings proved to be very effective for bringing to focus quickly many of the difficult 
problems that could only be resolved with the proper representation of both contractor 
and NASA personnel. 

As the Apollo Program progressed and assumed some measure of routine, many 
of the meetings were conducted by teleconference. It would have been desirable to 
have similar briefings with the GFE contractors, but it was considered impractical 
because of the large number of GFE procurements. In these cases, assessment of the 
R&QA activities continued to depend on status reports supplemented by contractor 
plant visits and frequent teleconferences. 


Reliability and Quality Information 

Background . - Procedures for the storage and retrieval of R&QA information 
were developed for the Apollo Program. Requirements for automated storage and re- 
trieval of failure information were established with Apollo contractors. These files 
were supplemented by a library of inspection records, test reports, and other docu- 
ments required by R&QA, engineering, and program management personnel. 

Experience in failure information . - Apollo spacecraft contractors were required 
to provide for the storage and retrieval of failure information on magnetic tape at their 
facilities and to transmit a duplicate tape to JSC. This procedure did not adequately 
support the spacecraft milestone reviews because the elapsed time from date of fail- 
ure to first printout at JSC frequently was several weeks, which was too late to permit 
the review and approval of the proposed corrective action by the responsible JSC per- 
sonnel. Therefore, a method for obtaining more current information was needed. The 
JSC procedure was changed, and the magnetic tape file was retained as a historical 
data file on failures. Requirements were revised for the submittal of failure informa- 
tion to JSC. The contractors were required to notify JSC within 24 hours after re- 
ceipt of failure notification by reliability if failures occurred during certification, 
acceptance vibration and acceptance thermal tests, catastrophic failures during de- 
velopment and acceptance testing, or failures that had a direct impact on vehicles in 
prelaunch checkout at KSC. Similar notification had to be provided for significant 
unsatisfactory conditions (fire, explosion) or ground support equipment (GSE) safety 
failures. A failure-information focal point (clearinghouse) was established by R&QA 
to receive and distribute failure reports at JSC (Houston) to the program office and to 
engineering. As a part of this clearinghouse activity, R&QA kept a current list of all 
unresolved (open) failures and related problems. Special emphasis was placed on 
early resolution of the problems affecting the vehicles at the launch site, using spe- 
cially prepared lists for frequent reviews with program office management. 
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Assessment of Reliability and Quality Assurance Functions 


Background. - The management of R&QA functions consisted of two principal 
activities. These activities were the establishment of R&QA requirements and, there- 
after, assessment of the functions and of the deliverable equipment to ensure that the 
activities were effective. The JSC assessment of R&QA functions was not a one-time, 
one-type action; rather, it was a continual series of planned and integrated actions. 
These actions include the following. 

1 . Audits of contractor and subcontractor R&QA functions and assessment of 
corrective action 

2. Management review meetings between JSC and contractor R&QA managers 
and supervisors and assessment of contractor responses to action items 

3. Weekly reports of significant events from the resident JSC R&QA personnel 
at prime contractor facilities 

4. Monthly reports from the JSC representative at subcontractor facilities 

5. Analyses of test and inspection records 

6. Reports generated by JSC R&QA in the performance of R&QA tasks 

Background - R&QA audits . - Audits of contractor and supplier R&QA activities 
to determine compliance to contractual R&QA requirements were performed by JSC 
as required by NASA policy and program directives. 

Experience . - Many contractors and suppliers providing space-flight equipment 
to NASA centers and to DOD agencies were audited several times each year. These 
independent audits by the centers and agencies created a problem with the contractors 
because many R&QA manpower resources had to be used to participate in the audit. 
This, in turn, affected the functions of the R&QA organization because key personnel 
supporting the audits were unable to concentrate on their day-to-day operations. To 
ensure that these audits would not be performed independently at least within NASA, 
the R&QA Office established a central focal point to coordinate audits with NASA 
Headquarters and its centers. This coordination included the establishment of sched- 
ules of the audits to be performed for the next 6 months and invited the other centers 
to participate and become members of the audit team. Before the audit, a determina- 
tion was made about which R&QA activities were to be audited, the type of checklists 
to be used, and how the audit would be conducted. Audits that are planned, scheduled, 
and coordinated have been beneficial to NASA and the centers in terms of cost effec- 
tiveness because of less travel expenditures and in terms of better contractor and 
supplier R&QA management because there was less disturbance in their operations. 

Assessments of Space-Flight Equipment 

Background . - Two major assessments or reviews of the individual Apollo space- 
craft were made by the program office before each launch. The first assessment, a 
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Customer Acceptance Readiness Review (CARR), was performed to assess the readi- 
ness for the initiation of individual subsystem, integrated testing, and Government 
acceptance of the spacecraft before shipment to KSC. The second assessment, a 
Flight Readiness Review (FRR), was to evaluate the spacecraft and related GSE before 
launch and to accomplish the mission. The R&QA personnel assisted in these assess- 
ments by being members of the data review teams and also as members of the formal 
CARR and FRR Boards. The purpose, composition of the review teams and formal 
boards, and procedures are defined in the program office CARR Plan and FRR Plan. 
The R&QA representatives documented the results of these reviews in the form of 
assessment statements signed by the Manager of the R&QA Office. These statements 
were presented to the CARR Board stating there were no constraints to shipment of the 
vehicle or to the FRR Board indicating that the spacecraft was ready for launch. In 
the event there was a constraint (such as open items), the assessment statement would 
indicate that at satisfactory completion of open items there would be no constraint. 

Experience . - Initially, formal reviews of spacecraft records were not performed 
before shipment of the spacecraft to the launch site. The reviews of vehicle records 
made by JSC were performed on a noninterference basis. For the early missions, the 
records and documentation were reviewed immediately before the FRR. The review 
of these records was concentrated mainly on nonconformance records and evaluation 
of the action taken to correct discrepancies; certification test reports; parts replace- 
ment records; and other inspection and test data, including nonmetallic materials 
(NMM) usage, EEE parts application, pressure vessel histories, and contamination 
and cleanliness records. As the program progressed, the amount of documentation 
increased significantly, and it was not possible to do this data and document review in 
the time allotted before the FRR. With the establishment of an additional program 
milestone review such as the CARR, the assessment procedures were revised to in- 
clude incremental reviews by R&QA at the milestone reviews, supplemented by con- 
tinual R&QA assessment of failures and problems. This procedure was much more 
effective in ensuring early detection and correction of problems that occurred after 
the incremental reviews, thereby permitting program management to be aware of the 
latest readiness status of the space-flight equipment and allowing adequate time for 
actions and corrective measures in the event of problems. 


Training 

Background . - Early in the Apollo Program, the fabrication and operations re- 
lated to special processes, particularly the soldering operations, were of concern to 
NASA Headquarters. Because of the unknown environments that the critical space- 
flight equipment may be subjected to, NASA considered it necessary to ensure that 
uniform soldering operations and techniques be used by industry fabricating this equip- 
ment. The NASA published a soldering requirements document (ref. 1) defining these 
operations and techniques. Because NASA as well as contractor personnel required 
training for these operations and techniques, soldering schools were established by 
NASA for the training and certification of contractor and supplier operators, inspec- 
tors, and instructors. In addition, NASA personnel were trained and certified, thereby 
enabling them to evaluate the contractor operations and techniques effectively. As the 
required skills were developed and the personnel were trained, the schools were dis- 
continued and the contractors were responsible for the training and certification of the 
soldering personnel. 
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Experience. - The NASA publication requirements included detailed "how to" 
techniques for solder operations, and the training of personnel in these techniques 
was extensive. As the Apollo Program progressed, many contractors requested 
waivers and deviations to the NASA document. These were normally granted when the 
contractor soldering operations and techniques proved to be equally reliable. These 
requested deviations revealed that the NASA how-to techniques were not flexible enough 
to permit the use of equally good techniques developed by contractors, thus causing an 
unnecessary increase in costs to retrain personnel. Review of the NASA publication 
by the NASA centers considered the recommendation that industry be permitted to use 
their process specifications and techniques. A revised NASA soldering document de- 
leting the rigid how-to techniques was published, and the quality and workmanship of 
equipment were maintained with reduced training costs not only for the contractors but 
also for NASA and delegated Government agencies. 

Contractors are required to document the soldering program including informa- 
tion regarding qualification of instructors, procedures for training, lesson plans, in- 
struction hours, and procedures for the certification and recertification of solder 
personnel. Early development and implementation of the training program and pro- 
cedures are requisites to ensure the availability and maintainability of reliable 
equipment. 


Design and Procedural Standards 

Significant spacecraft design and operational problems identified during Project 
Mercury and the Gemini Program were investigated to develop a clear and complete 
understanding of each problem and to determine if preventive requirements were 
technically feasible for spacecraft in future programs. Requirements considered 
feasible were developed into design standards, procedural standards, and engineering 
criteria bulletins. A Criteria and Standards Board was established with representa- 
tives from key JSC management and technical disciplines to review and recommend 
approval of the bulletins. After concurrence by the board, the bulletins were sub- 
mitted to the JSC Director for approval signature. After publication of the bulletins, 
the appropriate JSC elements and program office technical monitors were responsible 
for including the bulletin requirements in hardware procurement contracts. 

As the Apollo Program progressed, other significant problems were identified 
and investigated, and design and procedural standards (formerly bulletins) were pub- 
lished. When issued, standards were reviewed by the technical monitors to determine 
their applicability to active contracts. In many cases, the implementation costs pro- 
hibited retroactive application. When it was considered necessary to make retroactive 
application, the standards were imposed by Configuration Change Board action. Pro- 
cedures were developed whereby R&QA personnel in their review of new purchase 
requests verified if standards were included as requirements. If standards were not 
cited, the technical monitor was contacted to verify that the standards were not ap- 
plicable to the procurement. 

Experience at JSC indicated that, for new programs and contracts, it was neces- 
sary to have the current standards imposed in the statement of work, requiring the 
contractor to review and determine the degree of compliance with each standard. Ap- 
proval must be obtained from JSC for those standards not complied with by the 
contractor. 
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ENGINEERING FUNCTIONS 


The reliability of space-flight equipment is a function of the emphasis applied 
during equipment development by equipment designers, stress analysts, and other 
engineering personnel. The most effective contributions to product integrity are made 
by the engineer who appropriately considers the environments involved, selects the 
parts and materials, establishes the tolerances, and prepares the documents from 
which the product is purchased, fabricated, assembled, tested, and handled. Many of 
the NASA provisions for reliability and quality assurance, which are cited in the pub- 
lished NASA handbooks, can be implemented only by the equipment designers and 
engineers. 

Unreliable parts, use of incompatible materials, and insufficient attention to 
environments were a few examples of problems encountered during the program that 
clearly indicated a need for emphasizing the role of the designer in product integrity. 

Reliability Predictions 

Background. - Overall reliability numbers were generated as goals for the safety 
and success of the Apollo missions and were used as the basis for apportioning sub- 
system reliability goals. Initially, contractors were required to continue to develop 
reliability predictions and assessments as the Apollo design matured. 

Experience . - Rigorous numerical reliability predictions and assessments can 
be calculated only by using failure-rate information developed from actual hardware. 
Because of (1) the relatively limited amount of hardware developed for the Apollo Pro- 
gram, (2) the deletion of reliability testing, and (3) the relatively limited operating 
time accumulated on the hardware in actual environments, the requirement for the 
generation of rigorous reliability predictions and assessments was deleted. There- 
fore, the use of reliability numerical techniques has been relegated to use only in 
initial trade-off studies during the conceptual design phases of hardware development. 


Failure Mode and Effects Analysis 

Background . - As an integral part of the early spacecraft design phase, the flight 
hardware design should be analyzed to determine possible modes of failure and the ef- 
fects of failures on mission objectives and crew safety. Ideally, analyses should be 
conducted at the system, subsystem, and component levels including individual 
switches, relays, and so forth. The primary objective of these analyses is to identify 
the failures that could affect mission success or crew safety, so that redundant sys- 
tems can be added to the design where possible. 

Experience . - The spacecraft FMEA was an effective tool for design evaluation. 
Each potential failure was categorized for worst-case effect on mission success. For 
each case in which a single failure could cause a risk to the mission success or crew 
safety, the hardware in question was listed separately in an SFP list. Each portion of 
the design that had an SFP then got special attention, and the SFP was eliminated 
where possible. In many cases, the complete elimination of the bad effects of an SFP 
malfunction was not desirable because such action would make the normal function 
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more difficult to obtain. For example, the use of single switches to initiate parallel 
sequences of events with redundant crossovers that require precise timing and the use 
of series-parallel contacts for safety could have jeopardized normal sequencing. In 
such cases, procedures were devised to minimize vulnerability to inadvertent switch 
closure, such as keeping the system disarmed and using circuit breakers at all times 
except when arming was required. 

In other cases, any backup for an essential function was simply impractical, 
such as the case of the lunar module (LM) ascent stage engine used to leave the lunar 
surface. In such cases, extensive testing programs for qualification and flight engines 
were used, together with rigid configuration control and quality assurance surveillance 
of each flight article to minimize the risk. 

Although the FMEA and SFP information provided by contractors was usually in 
different formats and the data were presented in various manners, an analysis was 
accomplished. To overcome this problem and to ensure a uniformity of data on the 
spacecraft FMEA and SFP summaries, a recommended format was established for 
documenting the information (figs. 1 and 2). 


FAILURE-MODE AND EFFECTS ANALYSIS 
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Figure 1. - Format for failure mode and effects analysis. 
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SINGLE-FAILURE-POINT SUMMARY 


SYSTEM _ 
SUBSYSTEM 
ASSEMBLY 



Figure 2. - Format for single failure point summary. 


The FMEA and the SFP summaries were not required for all flight hardware. 

In some cases in which the hardware was not essential to the completion of the mission 
(for example, a simple experiment) and a malfunction could not be hazardous, the 
FMEA and the SFP summaries were omitted. 

The JSC experience with failure mode and effects analyses and SFP analyses 
indicates that these analyses are most effective when used in conjunction with design 
reviews. The Apollo spacecraft engineers performed comprehensive failure mode and 
effects analyses and SFP summaries and, through a series of iterative design reviews, 
eliminated or minimized the number of potential failure points (ref. 2). 

Electrical, Electronics, and Electromechanical Parts 

Background . - The original NASA provisions for control of EEE parts included 
the use of parts specialists to act as advisers to the design groups and to conduct the 
parts activities. The specific parts provisions included selection of parts, preparation 
of specifications, qualification tests, preparation of approved parts lists, and review of 
parts applications. 


CRITICALITY CATEGORY REFERENCE FMEA 

PREPARED BY 

APPROVED DATE 

SUPERSEDING 
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Experience . - These provisions were only partially effective. Design reviews 
conducted at the parts level disclosed several problems that required changes in the 
provisions to ensure the reliability of the Apollo equipment. A parts reliability require- 
ment document was developed by JSC that required that a parts program plan be devel- 
oped by contractors. This plan included requirements for development of specifications, 
one-way traceability, derating of parts, screening, and burn-in procedures and 
techniques. 

Review of the stress analyses for electronic assemblies revealed several cases 
in which the parts were stressed beyond the values established in the parts specifica- 
tions. Several failures of Apollo equipment were attributed to parts overstress. On 
the basis of this experience, the current goal is to have any new parts design stresses 
(voltage and so forth) no higher than 75 percent of the rated capability. This under- 
stressing is usually referred to as derating. 

The practices of screening parts for defects before use and burn-in (to eliminate 
infant mortality parts) were not implemented for some of the early Apollo equipment, 
and several failures of this equipment were traced to defective parts. In many cases, 
analysis disclosed that the failures could have been averted if screening and burn-in 
techniques had been used. As a result of these lessons, a dedicated effort was made 
to secure screened and burned-in parts for each critical flight application. 

Background - parts information . - An automated file of information on EEE parts 
was established at JSC. Parts usage information was obtained from contractors and 
suppliers and processed into the file, and reports were made and distributed to con- 
tractors. This allowed the contractors to compare parts test plans with the report in- 
formation to determine if the same or similar parts had already been tested by another 
contractor. More than $2 million in planned testing was avoided in this manner. 

Experience . - Initially, it was thought that the parts usage information would be 
extremely valuable to the designers of the next generation of space systems. However, 
it was soon realized that the rapid technological advances in the EEE parts field makes 
much of the information obsolete within a short time unless updated parts usage and ap- 
plication information is continuously inserted into the system. Therefore, the useful- 
ness of the file may be limited primarily to current programs. In addition, the file is 
useful for assessing the reliability of space-flight equipment at milestone reviews. For 
example, when equipment failures occur and the failure cause is isolated to an EEE part, 
the parts master file can be used to determine where the part type is used other than 
the failed equipment. The file is effective to a limited degree, but, like all automated 
files, it may not be current. In those cases when current data are missing, it becomes 
necessary to have the contractor search their records (such as material and parts lists) 
for the usage and application of suspect parts. However, the JSC file does permit the 
initial rapid location and investigation of the equipment. 


Nonmetallic Materials 

Background . - From a materials standpoint, a significant experience of the Apollo 
Program is the use of nonmetallic materials in the oxygen- rich spacecraft cabin atmos- 
phere. The reliability tasks contained in the NASA handbook "Reliability Program 
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Provisions for Aeronautical and Space Contractors” included general requirements re- 
garding materials but did not emphasize the importance of nonmetallic materials. 

Experience . - At the start of the spacecraft design effort, the amount of data on 
the flammability of NMM in pure oxygen at 5 psia and the toxicity of NMM was limited. 
Extensive testing of a variety of nonmetallic materials was required to find the least 
flammable and toxic material for each application. As the program progressed, many 
changes in materials were made as data were developed and requirements were included 
in contracts. The control of the NMM used in the Apollo spacecraft contributed largely 
to the enhancement of crew safety in the spacecraft. However, other important factors 
that must be considered for crew safety include control of ignition sources, control of 
the environment, fire-detection capability, and fire- extinguishment provisions. A sys- 
tems engineering approach to firesafety (ref. 3) must be established early in the devel- 
opment of space-flight programs. 

Offgassing, the term used to describe the evolution of gaseous products from liq- 
uid or solid material, represents one of the hazards associated with the use of NMM. 

In some cases, the offgassed products are toxic. In other cases, the offgassed products 
can coat sensitive equipment and degrade its performance. Although most organic ma- 
terials offgas to some extent at normal atmospheric conditions, many materials do not 
offgas significantly unless exposed to elevated temperatures, reduced pressures, or a 
combination of elevated temperature and reduced pressure. In the Apollo spacecraft, 
offgassing characteristics of materials must be evaluated during the material selection 
process. Material selection criteria are presented in reference 4. 

Background - materials information . - Materials tests were conducted at contrac- 
tor and JSC facilities. However, JSC soon realized that performing the tests at sev- 
eral different locations could result in duplicate testing and unnecessary expense. Data 
on the tests performed were accumulated and reproduced at JSC and distributed to con- 
tractors for review and analysis with the objective that contractors would not have to 
test materials already tested by other contractors. 

Experience . - The automated file of NMM information was designed to provide 
test data to contractors, Government organizations, and other organizations that wanted 
the data and also to yield NMM usage information for each Apollo spacecraft (material 
type, location in the spacecraft, weight, and so forth). Data from standardized tests 
on flame propagation, ignition temperature, and toxicity of offgassed products were 
accumulated and published periodically in JSC document MSC-02681, "Nonmetallic Ma- 
terials Design Guidelines and Test Data Handbook. ” The current issue of this document 
contains information on approximately 15 000 standard tests (conducted at six test facil- 
ities) that have been made on approximately 3700 different nonmetallic materials that 
are used or were considered for application on the Apollo spacecraft. Results from 
approximately 2600 special tests are also included in the document. The document pro- 
vided Government and industry personnel with test data and made possible the develop- 
ment of a cooperative program of data exchange between Government agencies and 
industry for the avoidance of duplicate testing. 
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Fluid Systems 

Background. - On a typical Apollo mission, 71 tanks are installed in the command 
and service module (CSM) and LM. These tanks contain items such as water, gaseous 
and liquid oxygen, liquid hydrogen, gaseous helium, gaseous nitrogen, nitrogen tetrox- 
ide, and Aerozine-50. The tanks have relatively thin walls and are highly stressed. 

For this reason, small surface imperfections can be significant stress risers and po- 
tential failure points. 

Experience . - Some of the more significant problems encountered in the fabrica- 
tion, installation, and testing of fluid systems are discussed in the following paragraphs. 

Pressure vessels . - The service pro- 
pulsion system (SPS) tanks installed in the 
Apollo service module are shown in fig- 
ure 3. The two helium tanks are spherical, 
are more than 40 inches in diameter, and 
are fabricated of titanium alloy. The oxi- 
dizer storage tank is 45 inches in diameter, 
is more than 153 inches long, and has a 
wall thickness of 0. 047 inch. The large 
size and thin walls make these tanks awk- 
ward to handle and thus extremely suscep- 
tible to inadvertent damage. 

Special handling procedures were de- 
veloped and used during the manufacturing, 
installation, and testing phases of these 
tanks, making all personnel aware of the 
critical operations associated with the han- 
dling of the pressure vessels. Experience 
related to pressure vessels used in the 
Apollo Program is discussed in detail in 
reference 5. Eighteen tank failures were 
reported during operations before flight. 

Some failures attributed to stress corro- 
sion, weld cracks, and materials defects 
are described as follows. 

Stress corrosion: The failures caused by stress corrosion were caused princi- 
pally by the use of fluids that contained substances that were incompatible with the tank 
materials. In one case, methanol was used as a test fluid for the SPS tanks. Analysis 
and testing in this case disclosed that a particular methanol and the titanium used are 
incompatible materials (ref. 6). In another case, impure water was used as a test 
fluid and failure resulted from incompatible materials. Experience with these problems 
showed that controls must be implemented to assure that fluids used in pressure vessels 
are compatible with the tank material. 

Weld cracks: An SPS propellant tank failed during an acceptance proof test; the 
failure was attributed to contamination in the weld joint between the cylindrical portion 
of the tank and the domed ends. Although the source of the contamination was not 
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Figure 3.- Service propulsion system 
tanks installed in the Apollo service 
module. 
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definitely established, it most likely occurred during the welding operation. Special 
controls were instituted to preclude recurrence of this problem including vapor blast 
cleaning before welding, control of the working and wetting environments, and the use 
of clean gloves during the welding process. These procedures apparently have been 
effective because no further problems have occurred with this type of weldment. 

Material defects: An LM descent propulsion system pressure vessel failed during 
a hydrostatic acceptance proof test. Investigation revealed that the failure was caused 
by localized microstructure abnormality, consisting of "massive" alpha-phase struc- 
ture in the upper dome. Alpha inclusions of this sort are rare and cannot be detected 
in the raw material with existing nondestructive testing techniques. Proof testing is 
the only practical approach for screening finished pressure vessels. 

The major nondestructive test methods implemented on the Apollo pressure ves- 
sels are ultrasonic inspection, X-ray photography, dye-penetrant inspection, and mag- 
netic particle inspection (steel tanks only). Nondestructive test methods have 
limitations. For example, X-ray photography can reveal surface and subsurface flaws 
but is sensitive to flaw orientation. Penetrant inspection is limited to the detection of 
surface flaws. These techniques must be used despite the limitations, and experience 
indicates that acceptance proof testing is the only way to ensure the integrity of the 
completed tanks. 

Background - pressure vessel historical records . - Every pressure vessel in the 
Apollo spacecraft was pressurized and vented many times before flight. The vessels 
were pressurized during acceptance tests by the suppliers and several times after in- 
stallation in the spacecraft. These pressurization cycles vary in pressure levels at- 
tained, elapsed time at pressure, fluids used, and temperature while under pressure. 

In addition, the tanks were frequently exposed to fluids for cleaning and purging. 

Experience . - Investigation of tank failures early in the program disclosed that 
detailed historical records had not been maintained for the individual tanks. Without 
such records, the f lightworthiness of each flight tank could not be determined because 
of lack of knowledge about pressure levels and cycles attained; duration at pressure 
and temperature; fluids used for fabrication, cleaning, and testing; and any noncon- 
formances recorded against the tank. Historical record cards were developed by JSC 
and requirements were imposed on the tank manufacturers and spacecraft contractors 
to maintain the cards current and to record the information and data just mentioned. 

Fluids . - The quality of the fluids used in the Apollo spacecraft systems was vital 
to the reliability of the systems; many failures of equipment were attributed to impuri- 
ties and contaminants in the fluids. In some cases, fluid system qualification tests 
were conducted with fluids that were of a higher purity (less contaminants) than the 
fluids used in acceptance tests or system operation. Many of the problems associated 
with fluids occurred from the lack of adequate standards or specifications for fluid 
cleanliness and fluid sampling. Guideline documents have been prepared to ensure 
proper control of fluids used in the spacecraft. Whenever changes are made to manu- 
facturing processes, handling procedures, or procurement sources for any fluids 
(liquids or gases), consideration should be given to requalifying a system that has been 
previously qualified using a particular fluid. 
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Contamination . - Contamination was one of the major problem categories on the 
Apollo spacecraft. The equipment contamination problems were attributable directly 
to inadequate knowledge in the disciplines required to maintain the level of cleanliness 
established during the equipment design phase. 

Basically, contamination in fluid systems can be categorized as follows. 

1. Fluids used for testing or operating the system; specifications or sampling 
procedures inadequate 

2. Fluid transmission systems (GSE and launch site facilities) not maintained to 
same level of cleanliness as the spacecraft 

3. Fluid system components not properly cleaned or sealed before installation 

4. Contaminants introduced into the system during system installation (brazing 
and soldering fluid lines, environment not controlled while fluid lines and components 
were being installed, and so forth) 

5. Fluid system components, such as quick disconnects, generating particles 
upon assembly or disassembly 

To prevent the occurrence of these problems, it was essential that controls in- 
cluding proper filtering be established at the start of fluid system design. The levels 
of cleanliness specified for the system by the designer must be maintained in the fluids, 
the system components, and the ground support equipment, at all locations. Controlled 
environments must be maintained during component assembly, installation, and removal. 
Plans and procedures were developed to ensure that the required disciplines were main- 
tained by all participants, including suppliers. 


Problem Control 

Background . - A system was developed by JSC for managing the disposition 
of problems (failures and anomalous performance) occurring on hardware during the 
Apollo Program. The system included all program participants (contractors, subcon- 
tractors, NASA test sites, and KSC) and provided for the recording and reporting of 
failures and other deficiencies, assessment of problems for cause and responsibility, 
corrective action to preclude recurrence, and screening operations that allowed prob- 
lems to be solved at the lowest management level. Only the most significant problems 
(high risk and unresolved) were presented to program management for review. The 
system was designed to support the spacecraft prelaunch reviews by providing summa- 
ries of subsystem problems occurring on the spacecraft in review as well as other 
problems that may affect the vehicle in review. The problem control system also was 
integrated with the NASA-wide ALERT system for dissemination of parts and materials 
problems. 

Experience - recording and reporting of problems. - A reasonable degree of 
awareness of problems and the impact of these problems on the program was difficult 
to maintain because of the large number of different problem reporting forms used by 
many contractors. Another difficulty developed concerning the report of failures to 
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JSC. The spacecraft contractors expressed concern about the definition of the term 
"failure" and the feasibility of reporting failures before acceptance test. To assure 
that all problems were properly documented and assessed, a failure was defined by 
JSC as "the inability of any part, component, or subsystem to perform its intended 
function under specified conditions and for a specified period of time" regardless of 
outside influence such as procedural or human error. All such problems were report- 
able to JSC after the beginning of hardware acceptance testing and required JSC ap- 
proval of the fixes by subsystem managers and R&QA personnel before the problems 
could be closed out. 

Problem assessment . - Some problems had their origin in the design and had to 
be solved by the designers. Other problems, which originated in the manufacturing ac- 
tivities, had to be corrected by the department responsible for the problem. Many 
problems of manufacturing origin were reported to JSC, where it was recognized that 
the investigation and closeout of this type of problem should be performed by contractor 
personnel at the contractor facility, with JSC personnel participating and concurring 
in the actions . 

This experience led to the establishment of problem assessment groups at the 
CSM and LM contractor facilities and at JSC. The prime function of the problem as- 
sessment groups was to review and screen the problem reports for manufacturing or 
design origin. All problems of design origin and significant manufacturing problems 
were transmitted to the JSC problem assessment group, where they were redistributed 
to the JSC subsystems managers, the reliability engineers, and the program office. 
Experience indicates that the problem assessment function was effective in prompt 
resolution and closeout of problems. 

Corrective action and closeout . - The ideal procedure was to review each problem 
and implement corrective action immediately. However, experience shows that this 
ideal condition could not be achieved. Discipline was required to assure that the most 
critical problems were addressed first and that the remaining problems were attacked 
on a priority basis. The contractor problem assessment groups were charged with the 
responsibility of assigning and approving corrective action. Final approval for close- 
out of problems of design origin was provided by the JSC subsystem manager and re- 
liability engineer. Closeout of problems of manufacturing origin was provided by the 
local quality personnel at the contractor plant or by JSC (Houston) quality assurance 
office, together with the appropriate subsystem manager. 

The NASA ALERT system . - Parts and materials application, failure, malfunc- 
tion, or problems of common interest were reported in the ALERT system. This 
NASA-wide system, established in 1964, provides for distribution of problem notices 
to NASA and contractors through coordinators at the various centers and to DOD through 
the Interservice Data Exchange Program (presently called Government-Industry Data 
Exchange Program (GIDEP)). The Apollo contractors were required to respond to JSC 
on each ALERT, stating whether or not the contractor hardware was affected. If the 
hardware was affected, a resolution to the problem was worked out. 
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MANUFACTURING FUNCTIONS 


General 

Analysis of the Apollo spacecraft failure records disclosed that a large percentage 
of the failures were caused by workmanship (10. 6 percent), contamination of equipment 
(8.2 percent), and other manufacturing causes (8.0 percent). Some of the more signif- 
icant experiences are discussed in the following paragraphs. 


Spacecraft Wiring 

Background. - Because of weight limitations, much of the wiring in the Apollo com- 
mand module was insulated with thin- walled Teflon covered with a thin polyimide coating. 
This wire was extremely susceptible to damage, and many special procedures were de- 
veloped to ensure the integrity of the wiring after installation in the spacecraft, during 
equipment installation and checkout, and during the mission. The complexity of the 
crew compartment wire harness is shown in figure 4 . 



Figure 4.- Crew compartment electrical harness. 
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Background - harness assembly and installation . - Because of the small size of 
the crew compartment and the limited number of technicians who can work in the crew 
compartment at the same time, it was impractical to try to do the spacecraft wiring 
(measuring, cutting, and other operations) within the compartment. 

Experience . - A special assembly fixture (fig. 5) was developed to assist in the 
assembly of the crew compartment harness. All fabrication operations were performed 
on the fixture, including installation of connectors. On completion of the harness as- 
sembly operations, an inspection was performed by the contractor, after which the 
harness was submitted to the Government inspectors for visual inspection. The final 
operation consisted of an automated air dielectric test while the harness was still in 
the assembly fixture. After completion of the harness checkout, the entire assembly 
was moved to the spacecraft assembly area for installation of the harness in the com- 
mand module. 






Installation of the crew compartment harness was a complex operation and re- 
quired considerable care to prevent damage to the harness. The completed harness, 
wrapped in a plastic material for protection, weighed more than 600 pounds and was 
extremely difficult to handle because of the many wire breakouts and connectors. The 
contractor devised a Teflon-coated trough (fig. 6) that fits through the command module 
hatch and allows the harness to be fed into the cabin. As the harness is fed into the 
command module, technicians position the wire bundle breakouts and clamp them into 
position. An example of wire routing and clamping is shown in figure 7. 



U it! 




Vv 



Figure 7.- Lunar module cabin wiring 
crimp (C) and solder (S) splices and 
end caps (EC). 



Figure 8.- Command module cable 
trays. 


Background - protection of wiring in- 
stallation . - Initially, the wiring in the crew 
compartment was exposed and subject to 
possible damage during ground-based oper- 
ations as well as during flight. 

Experience . - To protect the cables 
from physical damage and to reduce the 
risk of flame propagation, special protec- 
tive trays were designed (fig. 8). In addi- 
tion, these trays also are removable to 
permit changes in the wiring installation. 
Changes in wiring installations usually re- 
quire the mating and demating of connectors. 
To ensure that the connectors are not dam- 
aged, protective covers were needed and 
used. Metal screwcaps were used first, * 
but thread galling and metal particle con- 
tamination caused considerable concern. 

As a result, the following design and pro- 
cedural standard was prepared by JSC. 

Electrical plugs and receptacles of 
flight equipment and ground equipment that 
connect with flight equipment must be pro- 
tected at all times. Protective covers or 
caps are required over electrical plugs and 
receptacles whenever they are not connected 
to the mating part . The protective covers 
or caps must perform the following 
functions . 

1. Protect the plugs and receptacles 
from moisture 

2. Protect against damage to sealing 
surfaces, threads, or pins 

3. Be resistant to abrasion, chipping, 
or flaking 

4. Be brightly colored so as to be 
easily discernible and command attention 

5. Be maintained at a level of clean- 
liness equivalent to the plugs or receptacles 
on which they are used 

6. Be made of material that is com- 
patible with the connector material 
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Electrical Connectors 


Background . - Many electrical connectors are used in the spacecraft. Many of 
these connectors have a large number of closely spaced, small-diameter pins. Prob- 
lems encountered included crimping of small connector pins, moisture proofing the 
connectors, bent pins, and verification that connectors were mated properly. As these 
problems occurred, solutions were found and techniques and procedures were imple- 
mented to prevent recurrence. In some instances, design and procedural standards 
were developed and contractually imposed on contractors; such was the case regarding 
moisture proofing of connectors . The standard required that electrical connectors and 
wiring junctions to connectors be sealed from moisture to prevent open and short cir- 
cuits and that the composition and quality of the seal depend on the environment to which 
the seal will be subjected. 

Background - crimping . - The large incidence of crimp failures in the early days 
of Apollo manufacturing precipitated an intensive review of the crimping tools and the 
crimping techniques. One type of crimping tool then in use, even though properly cal- 
ibrated, frequently produced overcrimped pins, with the result that the wire would fail 
the pull test at the point of overcrimp. In other cases, wires were pulled from crimped 
pins because of improper calibration of the tool. 

Experience . - The crimping operations must be controlled rigidly to ensure that 
the proper tools, properly calibrated, are used by trained personnel. The following 
controls were established by the contractors. 

1. Each crimp tool was serialized. 

2. A calibration recall date was established for each tool. 

3. Each tool was preset and sealed for each crimp size. 

Despite these controls, a problem developed where 24-gage wires pulled out of 
connectors. Investigation disclosed that the crimps were made with the wrong tools. 
This condition occurred because the technician had several crimp tools at hand and 
selected the wrong one. The tool control procedures were changed to permit techni- 
cians to check out only one tool at a time. To verify that a crimp had been made prop- 
erly, test samples were made before and after each crimping operation and subjected 
to pull tests to verify that the proper tool was used and that the operator performed the 
crimping properly. 

Background - bent connector pins . - In many cases, the use of high-density con- 
nectors resulted in bent pins. Added to the basic problem of extremely delicate and 
easily bent pins was the frequent demating and mating or reconnection of connectors 
for equipment removal or installation. 

Experience. - The problem of bent connector pins became so severe that special 
training was provided for selected personnel, and, thereafter, all connector handling 
was performed by these trained personnel. Procedures for straightening pins were 
prepared, and requirements were established for recording all instances of bent pins. 
Experience indicates that constant vigilance must be maintained to ensure that connec- 
tor handling is a closely controlled function at all locations. 
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A technique used on the Apollo spacecraft to provide the capability of connector 
integrity control is to apply paint stripes on connectors that have been properly mated 
and verified to provide an indication when unauthorized demating of the connectors oc- 
curs. Special paint was required to ensure that the paint did not chip or flake and be- 
come a contaminant in the crew compartment. 

Radiographic inspection of electrical connectors . - Radiographic inspection was 
used to validate the integrity of certain connectors and terminal boards. While wire 
harnesses were still in the assembly fixture, electrical connectors were examined by 
X-ray photography. This procedure proved effective in detecting internal defects such 
as improper component seating, broken wires, bent pins, birdeaging, open joints, and 
conductive inclusions. Radiographic inspection of connectors and terminal boards also 
was used for troubleshooting after the wire harnesses were installed in the spacecraft. 
Small portable X-ray equipment made this procedure entirely practicable. In this man- 
ner, the connectors that could be X-rayed without having to remove other equipment 
could be inspected for bent pins and other defects. Experience with X-ray photography 
of electric connectors indicates that this practice should be used more extensively. The 
procedure was relatively inexpensive, and many defects that could not otherwise be de- 
tected were found with this method. 


Contamination Control 

Background. - The advent of manned space flight introduced stringent require- 
ments for cleanliness in the spacecraft, spacecraft subsystems, and ground support 
equipment. In the zero-g environment of space, loose objects in the spacecraft (nuts, 
bolts, washers) can be a nuisance and, in some cases, a hazard because they float 
freely about the interior of the spacecraft and can provide an unwanted conducting path 
in electrical circuitry or impede mechanical actuation of equipment. Experience with 
glass- column manometers and droplights used in the crew compartment indicated that 
breakage of these glass devices would constitute a threat to the safety of the crew if 
glass dust were left in the cabin and eventually inhaled by a crewmember. The use of 
all glass items in the spacecraft is restricted severely and controlled to prevent this 
hazard. Other forms of contamination included the spillage and leakage of fluids used 
in the spacecraft system. 

Experience - contaminants in fluid systems . - The fabrication and assembly of 
fluid systems and the associated components required precision cleaning techniques and 
controlled environments. Many of the experiences from the space programs are pre- 
sented in references 7 and 8. Many cases of fluid system contamination were caused 
by the introduction of contaminants during welding, soldering, brazing, or mechanical 
coupling of fluid system lines during installation of equipment. This experience led to 
the establishment of special procedures to minimize the chances of contaminating the 
systems during these operations. 

Experience - spacecraft cleanliness . - The following special controls were imple- 
mented to minimize the problem of loose materials in the crew compartment. 

1. The spacecraft was ’’tumbled” or rotated several times to recover loose arti- 
cles that otherwise were inaccessible. 
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2 . The crew compartment was vacuum cleaned frequently . 

3. Ultraviolet light inspection was used for detection of residual solvents and 
other potential contaminants. 

4. Filtered air was supplied to the crew compartment when the compartment 
was open. 

5. Hatch monitors were used to control entry into the crew compartment. Tools, 
materials, and equipment were recorded in a logbook during ingress and egress. 

6. Materials (cleaning fluids and so forth) that were used to service the space- 
craft were controlled and had to be approved before they could be near or in the 
spacecraft. 

Experience - fluid spillage . - Several incidences of fluid spillage occurred in the 
early phases of spacecraft assembly and test. For example, mercury was found in the 
crew compartment after a glass-column manometer shattered. The toxic effects of 
mercury on the crew or to spacecraft metals caused concern and resulted in the estab- 
lishment of design and procedural standards to preclude the use of mercury in or around 
the spacecraft. Another example of fluid spillage was the spillage of the coolant solu- 
tion (water/glycol) used in the environmental control system. Spillage of this material 
on spacecraft equipment and wire harnesses frequently necessitated removal and re- 
placement of the equipment and costly and time-consuming cleanup operations. Rework 
of the removed equipment to render it usable is also expensive. 

Tests were conducted to determine the effects of water/glycol solution on elec- 
trical wiring because the chemical reaction between the water/glycol and the silver- 
coated wire with broken insulation can provide a conductive path. Special procedures 
were developed to assure that water/glycol spills were cleaned properly with chemical 
tests to prove the adequacy of each cleaning. 

COMPONENT TESTS 
Background 

’’The single most important factor leading to the high degree of reliability of the 
Apollo spacecraft was the tremendous depth and breadth of the test activity. " The pre- 
ceding quotation is from reference 9 . 

Testing of the Apollo spacecraft equipment represents a substantial portion of the 
program resources and consists of two basic categories: certification testing and ac- 
ceptance testing. These test activities are discussed in the following paragraphs. 

Experience indicates that unless a comprehensive and well-integrated test plan 
is prepared, much unnecessary testing can be performed and some may be missed. Of 
greater importance from a reliability standpoint is the fact that testing may be per- 
formed at the wrong assembly level. This experience led to the development of special 
ground rules for testing Apollo spacecraft and equipment. These ground rules are dis- 
cussed in references 10 and 11. 
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Certification of Space-Flight Equipment 

Background. - The Apollo spacecraft equipment design is certified by test, by 
analysis, or by similarity to other certified equipment. This certification method 
minimizes the amount of hardware required for test purposes and shows the capabil- 
ity of the equipment to withstand mission environments and durations. More than 
700 tests were required before the first manned Apollo flight. 

Experience . - The certification testing activity was guided by the following rules. 

1. Use production equipment for test. 

2. Test at highest practicable level of assembly. 

3. Test all redundant paths. 

4. Test at or above maximum expected natural and induced environments. 

5. Perform acceptance test before certification test. 

6. Analyze to supplement testing. 

7. Understand and resolve all failures thoroughly. 

The distribution of failures detected during certification tests, acceptance tests, and 
flight is shown in figure 9. Reference 12 contains additional information about the JSC 
certification test program. 

Many design deficiencies were detected and corrected as a result of the certifica- 
tion test activities. A distribution of certification test failures by test environment is 
shown in figure 10. The dynamic environments yielded a substantial number of the total 
failures experienced. 



Acceptance Acceptance 


Figure 9.- Apollo spacecraft failures. 


Test environment 



Failures, percent 

Figure 10.- Distribution of Apollo 
spacecraft certification failures 
by environment. 
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Acceptance Test 


Acceptance tests were conducted on completed equipment and were intended to de- 
tect manufacturing defects. The acceptance test requirements for Apollo spacecraft 
equipment included thermal and vibration tests for those equipment items that were sus- 
ceptible to failure caused by temperature or vibration. Electronic equipment containing 
soldered connections is one example of this category. Defective solder connections can 
be detected during thermal and vibration testing, especially if instrumentation is used to 
monitor the circuitry for inter mi ttency during the test. 

Many manufacturing and design deficiencies can be detected during acceptance 
tests. A summary of acceptance test failures that occurred during vibration and ther- 
mal testing is contained in table I. The testing of spacecraft equipment is described in 
reference 11. 


TABLE I.- SUMMARY OF ACCEPTANCE TEST FAILURES FOR THE LM AND CSM BY INDIVIDUAL SUBSYSTEM 


Subsystem 

Construction type, 
percent 

Vibration tests 

Thermal tests 

Electronic 

Electro- 

mechanical 

Component 

types 

Units 

tested 

Number of 
failures 

Failures, 

percent 

Component 

types 

Units 

tested 

Number of 
failures 

Failures, 

percent 

Sequencers 

90 

10 

10 

169 

11 

6.5 

8 

159 

13 

6.2 

Environmental 

10 

90 

14 

339 

32 

9.4 

13 

197 

22 

11.2 

Electrical 

80 

20 

53 

847 

103 

12.2 

30 

449 

28 

6.2 

Stabilization 

80 

20 

11 

184 

28 

15.2 

11 

388 

71 

18.3 

Communication 

90 

10 

23 

441 

105 

23.8 

21 

273 

73 

26.0 

Instrumentation 

90 

10 

7 

118 

30 

25.4 

5 

469 

26 

5.6 

! Reaction control 

20 

80 

1 

543 

7 

1.3 

3 

757 

71 

9.3 

Propulsion 

15 

85 

9 

235 

16 

6.8 

3 

22 

1 

4.5 

Abort guidance 

80 i 

20 

9 

257 

17 

30.0 

5 

88 

100 

114.0 

Mechanical 

0 

100 

1 

74 

7 

10.6 

1 

51 

6 

11.7 

Radar 

80 

20 

4 

52 

32 

6.2 

0 

0 

0 

0 

Display components 

90 

10 

32 

7 928 

353 

4.5 

18 

848 

177 

21.0 

Display panels 

90 

10 

0 

0 

0 

0 

11 

106 

77 

7.3 

Total 

- 

- 

174 

11 187 

741 

6.7 

129 

3807 

665 

17.4 


Nondestructive Tests 

Background . - The conventional nondestructive methods (radiography, ultrasonics, 
dye penetrant, magnetic particle, and so forth) of testing equipment were used on Apollo 
spacecraft equipment. In some cases, combinations of these methods were used be- 
cause of the limitations of a particular method in a particular application. 
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Experience . - Multiple inspections using nondestructive test methods can be used 
effectively if there is doubt regarding the reliability of the equipment. For example, 

LM descent stage gaseous-oxygen tanks that had been proof pressure tested were re- 
called and subjected to additional nondestructive testing. These tanks were reinspected 
by using X-ray photography, ultrasonic pulse-echo, and dye-penetrant nondestructive 
methods and were found to be acceptable for flight use. Neutron radiography provided 
good results in detecting hydrogen embrittlement in weldments and for inspection of 
pyrotechnic devices. 

One innovation in radiographic inspection was very effective. Polaroid film was 
used with conventional portable X-ray equipment to "rough in" shots of items or objects 
in difficult areas. The use of this film saves much time because of the short develop- 
ment time compared to that of conventional X-ray film. 

Background - receiving inspection at KSC . - When the spacecraft arrived at KSC, 
an inspection was performed by the vehicle contractor and NASA, primarily as a check 
for damage in transit. Many deficiencies were found, some of which were not recorded 
at the time of vehicle shipment and others which had been recorded and accepted "as 
is. " Thus, it was found that the inspectors at the two sites, at KSC and at the contrac- 
tor facility, were inspecting without benefit of written acceptance instructions or crite- 
ria that were compatible with each other . 

Experience . - Inspection procedures were prepared, and a set of deficiency dia- 
grams was shipped with each vehicle to indicate where deficiencies had been noted and 
recorded at time of shipment. This condition is unusual because the spacecraft is 
shipped with many installations and operations that can be performed only at KSC . Ex- 
perience indicates that much unnecessary effort and paperwork can be averted by a co- 
ordinated planning activity of the inspection operations regardless of where they are 
performed. 


CONCLUDING REMARKS 


To be effective, reliability and quality assurance requirements and functions must 
be planned, monitored continually, and adjusted as the program develops and advances. 
The reliability and quality assurance activities are, in reality, disciplines that, if prop- 
erly implemented, will enhance the probability of producing and delivering reliable 
equipment. These disciplines have to be selected or tailored for each procurement. 

Some experiences related to the disciplines that were monitored and adjusted as the 
program developed are as follows. 

1. Specifying in contracts the scope, format, and content of reliability and quality 
assurance plans 

2. Establishment of customer/contractor reliability and quality assurance manage- 
ment review meetings of reliability and quality assurance program status 
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3. Development of incremental data and documentation reviews associated with 
program milestone reviews for assessment of space-flight equipment 

4. Application of broad specifications for soldering, welding, painting, and similar 
processes, instead of detailed step-by-step requirements 

Effective contributions to product integrity are made by engineering personnel 
who select parts and materials, establish tolerances, consider environments involved, 
and prepare the documentation from which the product is purchased, fabricated, and 
handled. Some experiences encountered in association with engineering functions that 
contributed to this product integrity are as follows. 

1. Documentation of failure mode and effects analysis and single failure point 
information to an established format 

2. Development of contractor electrical, electronic, and electromechanical parts 
program including requirements that parts be appropriately derated and screened before 
use and that burn-in procedures and techniques be used 

3. Controlling use of nonmetallic materials in spacecraft and for controlling 
cleanliness of equipment and fluid system 

The fabrication, assembly, and testing operations of spacecraft equipment during 
the manufacturing phase required that adjustment and changes be made to those opera- 
tions and that new techniques be adopted to preserve the reliability of the product. Ex- 
periences related to this phase of the program include the following. 

1 . Development of a special crew compartment harness fixture 

2. Development of design and procedural standards for the protection of electric 
connectors 

3. Establishment of controls for wire crimping operations 

4. Development of procedures and controls for the cleanliness of spacecraft 

5. Development and establishment of a comprehensive and well-integrated testing 
program 


Lyndon B. Johnson Space Center 

National Aeronautics and Space Administration 
Houston, Texas, July 12, 1973 
951-18-00-00-72 
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